Standing on the Shoulders of Giants Ranking by Combining Multiple Sources

In most ranking systems for information retrieval, there are multiple signals that need to be combined to generate a single score for ranking a result. In this work we consider how the output score of various ranking systems can be combined to rank the TREC 2003 and 2004 benchmark datasets. Not surprisingly, we find that the performance of a combined system is able to achieve significantly better performance than any single ranker. We also find that the best performing combination is that of a form of weighted sum of a link-based and text-based method, in which the text-based method is given most of the weight. In addition to combining the scores of multiple rankers directly, we also consider their use as features fed to a classifier. Interestingly, while classifiers perform classification poorly on the benchmark data, they can be used to benefit the ranking problem. We find that the scoring generated by a linear SVM classifier can perform respectably for ranking, and when used in combination with another ranking feature, can outperform all other tested ranking combinations.